MyOpenCRE mapping job: reuse the Librarian for user standards (stacked on #1171) - #1172

Fixes #965.

RFC user-auth-myopencre TODO 3, built on the maintainer decision recorded in #965: there is no second mapping stack. run_mapping takes a user standard's PENDING sections and runs them through the same C.1 retrieve, C.2 rerank, C.3 calibrate and C.4 decide the OIE pipeline runs, then records one outcome per section. The status rule is decision_engine.decide itself, not a re-derivation, so MyOpenCRE and the pipeline cannot disagree about what a confidence means.

Stacked on #1171 (branch link_suggest_275): it reuses that PR's build_retriever_for and candidate resolution, so this diff shows both commits until #1171 merges. Merge order: #1171 first, then this.

What this ships

Outcomes per section

• AUTO_MAPPED: confidence at or above the Librarian's link threshold (CRE_LIBRARIAN_LINK_THRESHOLD, 0.80 as held by the W7 sweep), cre_db_id set.
• NEEDS_REVIEW: below the threshold, with the candidate kept as the reviewer's hint; also where the graph could not resolve the candidate, then with no reference.
• UNMAPPABLE: nothing retrieved at all.
• USER_CONFIRMED / USER_REJECTED: never read, never written. set_section_mapping refuses to move them even if asked, and the refusal is in the UPDATE's WHERE clause, so a decision that lands between the job's read and its write still wins and the invariant does not depend on every caller remembering it. A re-index re-run (statuses=REMAP_AFTER_REINDEX) re-maps AUTO_MAPPED and NEEDS_REVIEW and still leaves them alone.

Design decisions worth a look

• Internal ids, validated. Retrievers return the embeddings pool's keys, which are the CRE rows' internal ids. Candidates resolve through the graph (the same _resolve_cre the suggest endpoint uses) and persist through get_cre_internal_id, so the stored reference is always a real row.
• Contained failures, idempotent re-runs, visible on the queue. A section whose stage raises stays PENDING and is counted in errored; the next run picks it up and processes nothing else twice. run_mapping_job raises MappingPartialFailure after the loop when errored > 0, so rq records the job as failed instead of reporting a clean run with leftovers; the sections that did map are already committed.
• A queue someone consumes. The default queue is default, which application/worker.py listens to out of the box; a test pins DEFAULT_QUEUE to worker.DEFAULT_LISTEN so the two cannot drift apart silently.
• Re-upload semantics. A changed section text returns to PENDING (and loses its stale mapping) unless the user has decided it; unchanged text is untouched. Returns rows touched.
• Worker-side, not web-side. The cross-encoder loads inside run_mapping_job; the web process gets shortlists from /rest/v1/link/suggest and never pays for the model.

Tests

19 hermetic tests (in-memory SQLite, stub components, no key, no model): every section starts PENDING; deterministic counts (4 sections -> 2 auto-mapped, 1 review, 1 unmappable); the stored cre_db_id is the validated internal id with the calibrated confidence beside it; an unresolvable candidate goes to review with no reference; a re-run processes nothing; USER_CONFIRMED survives a re-index re-run under changed confidences while undecided rows are re-mapped; an errored section stays PENDING; the threshold comes from the Librarian config; confirming a CRE not in the graph is refused; another user sees nothing and cannot decide; re-upload resets changed text but keeps user decisions; same name for two users is two rows and a duplicate for one user is refused; a repeated section id in one upload updates rather than double-inserts; a job write cannot overwrite a decision made after it loaded; confirming a different CRE clears the job's confidence; a partial run raises; the default queue is in the worker's listen list; an empty status list reads nothing.

Verification

• Migration applied through Alembic on a scratch SQLite (full chain to c4a1e7b2d9f0), downgraded to f3a91c07d24b (both tables gone), re-upgraded. Single head.
• db.py strict mypy error count: 129 on main, 129 on this branch; mapping_job.py strict-clean; black clean.
• Broad suite: 1049 passed; the four failures are the same pre-existing ones on main (dataset determinism, spreadsheet headers, two live-network tests).

Not in scope

• The HTTP upload route and the MyOpenCRE UI for review (RFC TODO 2's endpoints, TODO 4). The RFC says TODO 3 must be testable on fixtures before any upload exists, and it is.
• The automatic re-index trigger. REMAP_AFTER_REINDEX is the call for it; wiring it to the embeddings refresh is a one-line follow-up once that hook exists.
• The RFC's 98% exact-match target on an ASVS upload needs the cross-encoder and a valid embedding key, which this environment does not have; the harness for it is run_mapping on a fixture standard, unchanged.